{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kn1T9F5L4Hhy"
      },
      "source": [
        "# Profiling ONNX Models\n",
        "This notebook is a companion of chapter 9 of the \"Domain Specific LLMs in Action\" book, author Guglielmo Iozzia, [Manning Publications](https://www.manning.com/), 2024.  \n",
        "The code in this notebook is about profiling and getting performance insights for a [GPT-2 small]((https://huggingface.co/openai-community/gpt2) model after conversion to the [ONNX](https://onnx.ai/) format and optimization. The same code applies to any other LLM and the insights building part is generic for any ML/DL ONNX model profiling analysis. No hardware acceleration needed.  \n",
        "More details about the code can be found in the related book's chapter."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ePpgYStXAhul"
      },
      "source": [
        "Install the missing dependencies in the Colab VM (only ONNX and the ONNX runtime, plus mlprodict (for profiling data aggregation and clean up only). Please see note later in this notebook about the mlprodict package installation in later versions of the Colab runtime."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lzUHU9px7RlC"
      },
      "outputs": [],
      "source": [
        "!pip install onnx onnxruntime"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hNjxTAtDApie"
      },
      "source": [
        "Import the required packages and classes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OIASY74Orrbr"
      },
      "outputs": [],
      "source": [
        "import logging\n",
        "import numpy as np\n",
        "import torch\n",
        "from transformers import AutoTokenizer, AutoModelForCausalLM, BatchEncoding, GPT2LMHeadModel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jNM84XKAAswb"
      },
      "source": [
        "Download the GPT-2 small model and companion tokenizer from the HF's Hub."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "73EvwK9Y2m6C"
      },
      "outputs": [],
      "source": [
        "model_name = \"openai-community/gpt2\"\n",
        "\n",
        "model: GPT2LMHeadModel = AutoModelForCausalLM.from_pretrained(model_name)\n",
        "model.eval()\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
        "model.config.pad_token_id = tokenizer.eos_token_id"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f_7Grv-ZBHBx"
      },
      "source": [
        "Generate text to verify that the vanilla model has been downloaded and set up properly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rLEUGkYq2t6H"
      },
      "outputs": [],
      "source": [
        "sample_prompt = 'Here is some text to encode Hello World'\n",
        "inputs = tokenizer(sample_prompt, return_tensors=\"pt\")\n",
        "print(\"input tensors\")\n",
        "print(inputs)\n",
        "print(\"input tensor shape\")\n",
        "print(inputs[\"input_ids\"].size())\n",
        "\n",
        "with torch.no_grad():\n",
        "    outputs = model(**inputs)\n",
        "\n",
        "logits = outputs.logits\n",
        "print(\"output tensor\")\n",
        "print(logits)\n",
        "print(\"output shape\")\n",
        "print(logits.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MRbxmJh2Bt5L"
      },
      "source": [
        "Export the model to ONNX."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hEzctUUd3jTc"
      },
      "outputs": [],
      "source": [
        "input_ids: BatchEncoding = tokenizer(\n",
        "    sample_prompt, add_special_tokens=True,\n",
        "    return_attention_mask=False, return_tensors=\"pt\"\n",
        ")\n",
        "for k, v in input_ids.items():\n",
        "    input_ids[k] = v.type(dtype=torch.int32)\n",
        "input_tensor = input_ids['input_ids']\n",
        "\n",
        "onnx_model_path='gpt2onnx.onnx'\n",
        "torch.onnx.export(\n",
        "    model,\n",
        "    f=onnx_model_path,\n",
        "    args= (input_tensor,),\n",
        "    input_names=['input_ids'],\n",
        "    output_names=['logits'],\n",
        "    quantization=False,\n",
        "    var_output_seq=True,\n",
        "    do_constant_folding=True,\n",
        "    opset_version=18,\n",
        ")\n",
        "_ = model.eval()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KEWJNnIwB9Yo"
      },
      "source": [
        "Define a custom function to prepare the input for the ONNX model to run text generation for profiling."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V19WofplAc05"
      },
      "outputs": [],
      "source": [
        "def get_example_inputs(prompt_text, tokenizer, num_layer, device='cpu'):\n",
        "    encodings_dict = tokenizer.batch_encode_plus(prompt_text, padding=True)\n",
        "\n",
        "    input_ids = torch.tensor(encodings_dict[\"input_ids\"], dtype=torch.int32)\n",
        "    attention_mask = torch.tensor(encodings_dict[\"attention_mask\"], dtype=torch.int32)\n",
        "    position_ids = attention_mask.long().cumsum(-1) - 1\n",
        "    position_ids.masked_fill_(position_ids < 0, 0)\n",
        "    position_ids = position_ids.to(torch.int32)\n",
        "\n",
        "    empty_past = []\n",
        "    batch_size = input_ids.size(0)\n",
        "    sequence_length = input_ids.size(1)\n",
        "    past_shape = [2, batch_size, num_attention_heads, 0, hidden_size // num_attention_heads]\n",
        "    for i in range(num_layer):\n",
        "        empty_past.append(torch.empty(past_shape).type(torch.float32).to(device))\n",
        "\n",
        "    return input_ids, attention_mask, position_ids, empty_past"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w5djD_xrCJjm"
      },
      "source": [
        "Collect some vanilla model specs that are rquired for preparing the input for the ONNX version."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KPN1w6CDBEnJ"
      },
      "outputs": [],
      "source": [
        "num_layer = model.config.n_layer\n",
        "num_attention_heads = model.config.n_head\n",
        "hidden_size = model.config.n_embd"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kxu307dJCUed"
      },
      "source": [
        "Run text generation using the ONNX model with profiling enabled.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hjFo-ymc_8i9"
      },
      "outputs": [],
      "source": [
        "import onnxruntime\n",
        "\n",
        "tokenizer.pad_token = tokenizer.eos_token\n",
        "input_ids, attention_mask, position_ids, empty_past = get_example_inputs(['Here is some text to encode Hello World'], tokenizer, num_layer)\n",
        "\n",
        "so = onnxruntime.SessionOptions()\n",
        "so.enable_profiling = True\n",
        "session = onnxruntime.InferenceSession(onnx_model_path, so, providers=[\"CPUExecutionProvider\"])\n",
        "ort_inputs = {\n",
        "    \"input_ids\": np.ascontiguousarray(input_ids.cpu().numpy()),\n",
        "}\n",
        "ort_outputs = session.run(None, ort_inputs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Saxm2LdFH4V"
      },
      "source": [
        "Close the inference session and collect the profiling data. The `prof` variable contains the name of the generated JSON file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RukFJF7_HLTq"
      },
      "outputs": [],
      "source": [
        "prof = session.end_profiling()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jQYVqPvFv3M_"
      },
      "source": [
        "# Model Optimization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GEDfqWnQRgxn"
      },
      "source": [
        "Set up the logging level to see in the output which kind of optimizations are automatically applied."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_BsBfm2k4Qxk"
      },
      "outputs": [],
      "source": [
        "logging.basicConfig()\n",
        "logging.getLogger().setLevel(logging.INFO)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nltxuz9PFWxY"
      },
      "source": [
        "Optimize the model using the ONNX's native optimizer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jcBMyBt-v6TD"
      },
      "outputs": [],
      "source": [
        "from onnxruntime.transformers import optimizer\n",
        "\n",
        "onnx_optim_model_path=\"gpt2onnx-opt.onnx\"\n",
        "optimized_model = optimizer.optimize_model(onnx_model_path,\n",
        "                                           model_type='gpt2',\n",
        "                                           num_heads=num_attention_heads,\n",
        "                                           hidden_size=hidden_size,\n",
        "                                           use_gpu=False,\n",
        "                                           opt_level=1,\n",
        "                                           verbose=True)\n",
        "optimized_model.convert_float_to_float16()\n",
        "optimized_model.save_model_to_file(onnx_optim_model_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JbyTg-toFhcR"
      },
      "source": [
        "Run text generation using the ONNX optimized model with profiling enabled."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KgTqn1IJN9Ds"
      },
      "outputs": [],
      "source": [
        "import onnx\n",
        "\n",
        "optimized_onnx_model = onnx.load(onnx_optim_model_path)\n",
        "\n",
        "tokenizer.pad_token = tokenizer.eos_token\n",
        "input_ids, attention_mask, position_ids, empty_past = get_example_inputs(\n",
        "    ['Here is some text to encode Hello World'], tokenizer, num_layer)\n",
        "\n",
        "so = onnxruntime.SessionOptions()\n",
        "so.enable_profiling = True\n",
        "session = onnxruntime.InferenceSession(onnx_optim_model_path, so,\n",
        "                                       providers=[\"CPUExecutionProvider\"])\n",
        "ort_inputs = {\n",
        "    \"input_ids\": np.ascontiguousarray(input_ids.cpu().numpy()),\n",
        "}\n",
        "ort_outputs = session.run(None, ort_inputs)\n",
        "prof_optimized = session.end_profiling()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pnet0VAXFqFF"
      },
      "source": [
        "# Profiling Data Clean Up and Visualization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CNq6hwS1OGSw"
      },
      "source": [
        "Copying and pasting here the original *mlprodict*'s `OnnxWholeSession` class code as the installation of this package is failing on the latest version of the Colab runtime."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tGvuiu1GPCgW"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "import numpy\n",
        "\n",
        "class OnnxWholeSession:\n",
        "    \"\"\"\n",
        "    Runs the prediction for a single :epkg:`ONNX`,\n",
        "    it lets the runtime handle the graph logic as well.\n",
        "\n",
        "    :param onnx_data: :epkg:`ONNX` model or data\n",
        "    :param runtime: runtime to be used, mostly :epkg:`onnxruntime`\n",
        "    :param runtime_options: runtime options\n",
        "    :param device: device, a string `cpu`, `cuda`, `cuda:0`...\n",
        "\n",
        "    .. versionchanged:: 0.8\n",
        "        Parameter *device* was added.\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, onnx_data, runtime, runtime_options=None, device=None):\n",
        "        if runtime not in ('onnxruntime1', 'onnxruntime1-cuda'):\n",
        "            raise NotImplementedError(  # pragma: no cover\n",
        "                f\"runtime '{runtime}' is not implemented.\")\n",
        "\n",
        "        from onnxruntime import (  # delayed\n",
        "            InferenceSession, SessionOptions, RunOptions,\n",
        "            GraphOptimizationLevel)\n",
        "        from onnxruntime.capi._pybind_state import (  # pylint: disable=E0611\n",
        "            Fail as OrtFail, InvalidGraph as OrtInvalidGraph,\n",
        "            InvalidArgument as OrtInvalidArgument,\n",
        "            NotImplemented as OrtNotImplemented,\n",
        "            RuntimeException as OrtRuntimeException)\n",
        "\n",
        "        onnx_data0 = onnx_data\n",
        "        if hasattr(onnx_data, 'SerializeToString'):\n",
        "            onnx_data = onnx_data.SerializeToString()\n",
        "        if isinstance(runtime_options, SessionOptions):\n",
        "            sess_options = runtime_options\n",
        "            session_options = None\n",
        "            runtime_options = None\n",
        "        else:\n",
        "            session_options = (\n",
        "                None if runtime_options is None\n",
        "                else runtime_options.get('session_options', None))\n",
        "            self.runtime = runtime\n",
        "            sess_options = session_options or SessionOptions()\n",
        "        self.run_options = RunOptions()\n",
        "        self.run_options.log_severity_level = 3\n",
        "        self.run_options.log_verbosity_level = 1\n",
        "\n",
        "        if session_options is None:\n",
        "            if runtime_options is not None:\n",
        "                if runtime_options.get('disable_optimisation', False):\n",
        "                    sess_options.graph_optimization_level = (  # pragma: no cover\n",
        "                        GraphOptimizationLevel.ORT_ENABLE_ALL)\n",
        "                if runtime_options.get('enable_profiling', True):\n",
        "                    sess_options.enable_profiling = True\n",
        "                if runtime_options.get('log_severity_level', 2) != 2:\n",
        "                    v = runtime_options.get('log_severity_level', 2)\n",
        "                    sess_options.log_severity_level = v\n",
        "                    self.run_options.log_severity_level = v\n",
        "        elif runtime_options is not None and 'enable_profiling' in runtime_options:\n",
        "            raise RuntimeError(  # pragma: no cover\n",
        "                \"session_options and enable_profiling cannot be defined at the \"\n",
        "                \"same time.\")\n",
        "        elif runtime_options is not None and 'disable_optimisation' in runtime_options:\n",
        "            raise RuntimeError(  # pragma: no cover\n",
        "                \"session_options and disable_optimisation cannot be defined at the \"\n",
        "                \"same time.\")\n",
        "        elif runtime_options is not None and 'log_severity_level' in runtime_options:\n",
        "            raise RuntimeError(  # pragma: no cover\n",
        "                \"session_options and log_severity_level cannot be defined at the \"\n",
        "                \"same time.\")\n",
        "        providers = ['CPUExecutionProvider']\n",
        "        if runtime == 'onnxruntime1-cuda':\n",
        "            providers = ['CUDAExecutionProvider'] + providers\n",
        "        try:\n",
        "            self.sess = InferenceSession(onnx_data, sess_options=sess_options,\n",
        "                                         device=device, providers=providers)\n",
        "        except (OrtFail, OrtNotImplemented, OrtInvalidGraph,\n",
        "                OrtInvalidArgument, OrtRuntimeException, RuntimeError) as e:\n",
        "            raise RuntimeError(\n",
        "                \"Unable to create InferenceSession due to '{}'\\n{}.\".format(e)) from e\n",
        "        self.output_names = [_.name for _ in self.sess.get_outputs()]\n",
        "\n",
        "    def run(self, inputs):\n",
        "        \"\"\"\n",
        "        Computes the predictions.\n",
        "\n",
        "        @param      inputs      dictionary *{variable, value}*\n",
        "        @return                 list of outputs\n",
        "        \"\"\"\n",
        "        v = next(iter(inputs.values()))\n",
        "        if isinstance(v, (numpy.ndarray, dict)):\n",
        "            try:\n",
        "                return self.sess._sess.run(\n",
        "                    self.output_names, inputs, self.run_options)\n",
        "            except ValueError as e:\n",
        "                raise ValueError(\n",
        "                    \"Issue running inference inputs=%r, expected inputs=%r.\"\n",
        "                    \"\" % (\n",
        "                        list(sorted(inputs)),\n",
        "                        [i.name for i in self.sess.get_inputs()])) from e\n",
        "        try:\n",
        "            return self.sess._sess.run_with_ort_values(\n",
        "                inputs, self.output_names, self.run_options)\n",
        "        except RuntimeError:\n",
        "            return self.sess._sess.run_with_ort_values(\n",
        "                {k: v._get_c_value() for k, v in inputs.items()},\n",
        "                self.output_names, self.run_options)\n",
        "\n",
        "    @staticmethod\n",
        "    def process_profiling(js):\n",
        "        \"\"\"\n",
        "        Flattens json returned by onnxruntime profiling.\n",
        "\n",
        "        :param js: json\n",
        "        :return: list of dictionaries\n",
        "        \"\"\"\n",
        "        rows = []\n",
        "        for row in js:\n",
        "            if 'args' in row and isinstance(row['args'], dict):\n",
        "                for k, v in row['args'].items():\n",
        "                    row[f'args_{k}'] = v\n",
        "                del row['args']\n",
        "            rows.append(row)\n",
        "        return rows\n",
        "\n",
        "    def get_profiling(self):\n",
        "        \"\"\"\n",
        "        Returns the profiling informations.\n",
        "        \"\"\"\n",
        "        prof = self.sess.end_profiling()\n",
        "        with open(prof, 'r') as f:\n",
        "            content = f.read()\n",
        "        js = json.loads(content)\n",
        "        return OnnxWholeSession.process_profiling(js)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "94AhN1kvGGLE"
      },
      "source": [
        "Define a custom function to put the raw ONNX profiling data in a more friendly and useful format."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pZgmisDGw1Ip"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "import pandas as pd\n",
        "\n",
        "def clean_up_profiling_data(prof):\n",
        "  with open(prof, \"r\") as f:\n",
        "      js = json.load(f)\n",
        "  df = pd.DataFrame(OnnxWholeSession.process_profiling(js))\n",
        "\n",
        "  return df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6fBgZH6kG5oA"
      },
      "source": [
        "Define a custom function to do several profiling data aggregations (group by operator type and calculate the total duration for each one, count the number of occurrences for each one (and order them by duration), calculate the percentage of the total inference time for each one) that would be used to build some visualizations."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yzVMU_34JFsN"
      },
      "outputs": [],
      "source": [
        "def transform_profiling_data_for_visualization(df):\n",
        "  gr_dur = df[['dur', \"args_op_name\"]].groupby(\"args_op_name\").sum().sort_values('dur')\n",
        "\n",
        "  gr_n = df[['dur', \"args_op_name\"]].groupby(\"args_op_name\").count().sort_values('dur')\n",
        "  gr_n = gr_n.loc[gr_dur.index, :]\n",
        "\n",
        "  gr_dur_perc = gr_dur / gr_dur['dur'].sum()\n",
        "\n",
        "  return gr_dur, gr_n, gr_dur_perc"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ek7GzdqZJz-l"
      },
      "source": [
        "Transform the profiling data for the ONNX model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AmXh-UjzKDMX"
      },
      "outputs": [],
      "source": [
        "gr_dur, gr_n, gr_dur_perc = transform_profiling_data_for_visualization(clean_up_profiling_data(prof))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vupd5KSdKO6W"
      },
      "source": [
        "Create visualizations for the ONNX model profiling data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uec6ve0JKVOZ"
      },
      "outputs": [],
      "source": [
        "import plotly.express as px\n",
        "\n",
        "fig = px.bar(gr_dur, x='dur',\n",
        "             labels={\n",
        "                     \"dur\": \"Duration (ms)\",\n",
        "                     \"args_op_name\": \"Operation type\",\n",
        "                 },\n",
        "             title='Duration')\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "osIk4V4GKfBx"
      },
      "outputs": [],
      "source": [
        "fig = px.bar(gr_n, x='dur',\n",
        "             labels={\n",
        "                     \"dur\": \"Op count\",\n",
        "                     \"args_op_name\": \"Operation type\",\n",
        "                 },\n",
        "             title='Occurrences')\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a2LpQbd5Kinz"
      },
      "outputs": [],
      "source": [
        "fig = px.bar(gr_dur_perc, x='dur',\n",
        "             labels={\n",
        "                     \"dur\": \"Duration (%)\",\n",
        "                     \"args_op_name\": \"Operation type\",\n",
        "                 },\n",
        "             title='Proportion')\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eGDjmfkTKths"
      },
      "source": [
        "Transform the profiling data for the optimized ONNX model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3BgEdxklKxJ2"
      },
      "outputs": [],
      "source": [
        "gr_dur, gr_n, gr_dur_perc = transform_profiling_data_for_visualization(clean_up_profiling_data(prof_optimized))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OhKzBjVGPpqV"
      },
      "source": [
        "Create visualizations for the optimized ONNX model profiling data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tcU81S43Pw1x"
      },
      "outputs": [],
      "source": [
        "fig = px.bar(gr_dur, x='dur',\n",
        "             labels={\n",
        "                     \"dur\": \"Duration (ms)\",\n",
        "                     \"args_op_name\": \"Operation type\",\n",
        "                 },\n",
        "             title='Duration')\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TGE1-FxRP0H-"
      },
      "outputs": [],
      "source": [
        "fig = px.bar(gr_n, x='dur',\n",
        "             labels={\n",
        "                     \"dur\": \"Op count\",\n",
        "                     \"args_op_name\": \"Operation type\",\n",
        "                 },\n",
        "             title='Occurrences')\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IFxzpi9QP4A-"
      },
      "outputs": [],
      "source": [
        "fig = px.bar(gr_dur_perc, x='dur',\n",
        "             labels={\n",
        "                     \"dur\": \"Duration (%)\",\n",
        "                     \"args_op_name\": \"Operation type\",\n",
        "                 },\n",
        "             title='Proportion')\n",
        "fig.show()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
